Eye-gaze tracking

ABSTRACT

A device and a method for tracking an eye-gaze of an observer. A deep blue or violet light source is used to emit light to eye, particularly to the retina. The deep blue light is partially reflected and partially absorbed by the retina. The absorption is most prominent around the fovea, the area of sharp vision, because of the pigment which protects the fovea from short wavelength radiation. Thus the device and method of tracking eye-gaze according to the invention comprises emitting light having a certain wavelength and transferring the light to the retina of an eye. The wavelength of the light being such as to make the fovea of the eye resolvable. The method further comprises detecting light that is reflected from the eye to form detection information including the resolvable fovea, and mapping the detection information to a predetermined surface, the surface being located at a distance from the eye, the location of the fovea on the surface forming an eye-gaze point.

FIELD OF THE INVENTION

This invention relates to eye-gaze tracking and, more particularly to aneye-gaze tracking device and method preferably for tracking the eye-gazeof a user on a surface such as a display.

BACKGROUND OF THE INVENTION

Keyboards, mice or joysticks provide a communication interface between ahuman and an electronic device. Eye-gaze measurements have been used inphysiological and psychological studies disclosed in document Arne JohnGlenstrup and Theo Engell-Nielsen: BS thesis, Laboratory of Psychology,University of Copenhagen which is available in Internet at the URLhttp://www.diku.dk/˜panic/eyegaze/article.html. Further eye-gaze tellsabout an interest area of an observer.

Present eye trackers are based on reflections of Infra Red (IR) lightfrom the different layers of the eye, which is known as Purkinjereflections, and on reflection from a retina, which is seen as a brightiris. There are trackers that use only Purkinje images and trackers thatcombine the reflections from cornea and bright iris. Those reflectionsmove with respect to each other and with respect to the bright irisdepending on gaze direction. Usually one IR point source is enough butin order to increase accuracy several IR sources have been used. Theretina has very large reflection at red and even more at IR wavelengths.Also the visibility of details, e.g. blood vessels, vanish when usinglonger wavelengths. Thus the retina looks very uniformly illuminated.This is why the iris-cornea method uses IR wavelengths.

Present eye trackers face major drawbacks to be able to fulfill the needfor general usage. Eye trackers that are based on the Purkinje and/orthe so called iris-cornea method must be calibrated frequently and acalibration is user dependent. In addition of that, these devices arenot suitable for all people, because the eye structure of some people isincompatible for the device. This is because of physiologicaldifferences in the eye, especially physiological differences in eyelidpositions. Other concerns are slowness of the device, that is, a delayin the existing devices prevents the required control.

An additional drawback is a general eye tracking structure of presenteye trackers. Present eye trackers are bulky and heavy on a systemlevel. These eye trackers require systems which are too massive forintelligent integrated electronic devices such as displays, virtualreality computer displays, portal computers and mobile phones.

Present eye-gaze tracking methods monitor the outer parts of an eye.These methods are very much person dependent and they are too muchaffected by personal eye geometry or the position of eyelids. Thuspersonal differences of eye physiology set a restriction for a commonusage. These methods are inaccurate for required controlling andtracking.

A need for an improved user interface setting hands free is evident, asthe devices become smaller, portable, more intelligent and ubiquitous.Eye-gaze tracking should set a data stream delivering high informationcontent to the part of a display the eye is gazing. Efficient monitoringof eye-gaze is inevitable to eye-gaze controlled communication between ahuman and an electronic device. A control set could be defined byeye-gaze itself or eye-gaze combined to other existing control sets,such as a button. In addition, an eye-gaze tracking/control deviceshould provide a response with an imperceptible delay for a device userfor required usage. Current devices do not meet these abilities or theyprovide infeasible implementations.

SUMMARY OF THE INVENTION

The present invention provides a device and a method for tracking aneye-gaze on a surface such as a display of an electronic device and thusprovides a base for controlling the electronic device according to theeye-gaze of a user.

According to a first aspect of the invention there is provided a methodof tracking eye-gaze, the method comprising

emitting light having a certain wavelength;

transferring the light to the retina of an eye;

the wavelength of the light being such as to make the fovea of the eyeresolvable;

detecting light that is reflected from said eye to form detectioninformation including the resolvable fovea;

mapping the detection information to a predetermined surface, saidsurface being located at a distance from said eye, the location of thefovea on said surface forming an eye-gaze point.

According to a second aspect of the invention there is provided aneye-gaze tracking device, comprising:

a light source for emitting light having a certain wavelength;

means for transferring the light to the retina of an eye, the wavelengthof the light being such as to make the fovea of the eye resolvable;

a detector for detecting light that is reflected from said eye to formdetection information including the resolvable fovea;

a surface located at a distance from said eye; and

means for mapping the detection information to the surface for locatingthe fovea on said surface for forming an eye-gaze point by the locationof the fovea.

In a preferred embodiment of the invention a deep blue or morepreferably violet light source is used to emit light into the eye,particularly to the eye retina. Accordingly this means light having awavelength in the range of about 395-500 nm, where a violet light havinga wavelength of about 395-430 nm is preferred but blue light up to awavelength of about 480 nm also works. Further an optical x-y matrixdetector is used to measure the light reflected from eye retina. Fromthe reflected light the foveal position, which is the eye-gaze, withrespect to the optical assembly is measured with the detector.

According to another embodiment of the invention an eye-gaze tracker isintegrated to a display, which can be a virtual reality display.

In one embodiment of the invention, eye-gaze control can set a datastream delivering high information content to the part of a display thatthe eye is gazing. Also in an embodiment, a control set or a controlmaster is eye-gaze combined to hand controlling equipment. According toan embodiment of the invention, there may be provided software whichmeasures and calculates fovea from data received from the detector,preferably the x-y detector, which may be a CCD (Charge Coupled Device)for example. In a further embodiment, the invention is provided with apattern recognition calculation of blood vessels and comparing thecalculations with a calibration pattern of blood vessels.

The invention provides several advantages over prior solutions. Forexample, a device according to one method of the invention may beimplemented to be a simple light eye-gaze tracking device. This enablesan embodiment for Virtual Reality (VR) displays having a reasonablephysical size. Moreover, for portable computers or mobile units, such asmobile phones, a device for eye-gaze tracking according to the inventionof reasonable structure can be implemented.

Preferably tracking and detecting the eye-gaze is based on theobservation of the retina. This follows that the invention functions fordifferent eyes although personal physical eye structure may varyconsiderably. This allows a large amount of users to apply theinvention. The invention is thus suitable for substantially all usershaving normal physical human eye structure.

Moreover, the observation of the retina gives reliable and preciseinformation of eye-gaze. Eye-gaze direction can preferably be determinedwithout calibration of the device. Thus it is possible to obtaineye-gaze position on a display within immediate reaction from theeye-gaze device having measured with the detector (such as a CCDdetector) and mapped to a display, e.g. a LCD.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosed invention will be described with reference to theaccompanying drawings, which show embodiments of the invention andwhere:

FIG. 1 depicts a block diagram of the device architecture of a preferredembodiment,

FIGS. 2a and 2 b depict an elucidation for utilization about physicalfeature of retina of healthy eye at IR and at blue/violet wavelengths,

FIG. 3 depicts a flow chart of a preferred embodiment of a calibrationfree eye-gaze tracking,

FIG. 4 depicts a flow chart of another embodiment of the eye-gazetracking, and

FIG. 5 depicts a typical use situation of the eye tracking device of thepresent invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The invention will be described with particular reference to a preferredembodiment. However, it should be understood that the differentembodiments provide only a few examples of the many advantageous uses ofthe innovative teachings herein.

Various embodiments of the disclosed method and device will be describedutilizing the physical feature of the retina. FIGS. 2a and 2 b depict anelucidation for utilization about the physical feature of a retina of ahealthy eye at IR (infrared) light and at blue light or violet lightwavelengths, respectively. FIG. 2a shows the retina of a healthy eyeilluminated at IR light wavelengths. The retina has very largereflection at red and even more in IR wavelengths. Also the visibilityof details, e.g. blood vessels, vanish when using longer wavelengths. Onthe other hand when the wavelength is shortened, more and more detailscome out. Not only blood vessels but also fovea emerges. This can beseen from FIG. 2a where at IR light the fovea cannot be detected. FIG.2b shows the retina of a healthy eye illuminated at blue wavelengths. Atblue light the fovea emerges as a dark spot and the further we go toshorter wavelengths the more prominent the fovea is. Thus a deep blue orviolet light sets a clear and prominent fovea. This is due to the UVprotective pigment (xantofyllein-lutein), which is densest just in thefovea. The deep blue or violet light is partially reflected andpartially absorbed by the retina. The absorption is most prominentaround the fovea, the area of sharp vision, because of the pigment whichprotects the fovea from short wavelength radiation. This yellowpigmented area is called the macula lutea. Because of the absorption(whereas other parts of the eye reflect light at this wavelength) thefovea becomes resolvable as a dark spot. Accordingly this means lighthaving a wavelength in the range of about 395-500 nm. A violet lighthaving a wavelength of about 395-430 nm is preferred. When a light oflonger wavelength is used the fovea does not anymore emerge very clear.The fovea has been detected by blue light having a wavelength of about480 nm. Thus even a wavelength of 500 nm and a bit above still bringsout the fovea but when moving closer to red light (600-700 nm) the foveacan no longer be detected as shown in FIG. 2a. The fovea does appearclearly also for light having a wavelength less than 395 nm, but that isnot recommended as it may damage the eye. Thereby light with awavelength of 395-430 nm is preferred, and good results have beenachieved with 405 nm.

The eye tracking optics comprises a narrow band 118 beam splitter(implemented as a coating) and a deep blue or violet source 116 forillumination of the retina 100. In two passes through the quarter waveplate 122 (which also is narrow band, e.g. in the range of 405 nm), thepolarization of the short wavelength light will rotate by 90 degrees butdoes not affect the other bands of spectrum (except the band of thequarter wave plate). The LCD panel 108 does not alter the polarizationstate of this short wavelength light. The illumination optics 114,polarizer 120, prisms 124 and 126 and the quarter wave plate 122 work sothat the short wavelength light is guided to the retina 100. The shortwavelength light, which is reflected from the retina, is going back tothe device, but now the narrow band reflector 118 guides the lighttowards the detector 106. The focal plane of this light is inside theprism 126, and a relay lens 128 transforms the image from this planeonto the detector 106. The relay lens also takes care of correctscaling, because the display panel 108 and the detector 106 can havedifferent dimensions. The detector 106 can be for example a CCD cameraand is used to convert optical information to electronic or binaryinformation. The LCD panel 108 is a reflective microdisplay and is usedas a traditional displaying unit and as a function unit to set alocation to eye-gaze. The display panel 108 can be also transmissive butin this case the illumination is assembled behind the panel. The lightsource can be implemented as a LED (Light Emitting Diode) and also theother optics shown in FIG. 1 (i.e. the parts except the display) can bemade of small size and thus this may made in a size such as 18 mm×18mm×6 mm.

In the embodiment of FIG. 1, the imaging optics 104 transforms andtransfers a picture of the LCD panel to a picture on the retina 100.This is done to achieve a clear picture on the retina 100. Thus theimaging optics 104 and also the lens of the eye 102 sets rays from theLCD panel 108 to the retina 100 in such a way that the picture on theretina 100 is visible and noticeable according to its source from theLCD panel 108. The illumination optics 114 is formed so that the LCDpanel 108 has a homogeneous illumination and also so that the amount ofshort wavelength light 116 entering the eye is optimized, e.g., a safeamount of light that enters eye that causes no physical harm or damageto the eye. The short wavelength of light (˜400 nm or at least less than500 nm) can be so low of intensity or so short pulses used, that thehuman eye cannot resolve the light. Thus the eye tracking operation isnot observed by the user whereas the detector can be built verysensitive to this part of light spectrum. Thus the light source 112illuminates the LCD panel 108 homogeneously and has no considerableinterference to the detection at the detector 106 about light reflectedfrom the retina 100 to the detector 106. Also on the other hand, theshort wavelength source 116 has no considerable interference to raysforming picture emitted by the LCD panel 108 to the retina 100. The deepblue or violet illumination is arranged so that it visible when the LCDpanel 108 is homogeneously illuminated by deep blue or violet light. Theimage of retina is thus transformed on to the detector 106. There willbe some stray light from the cornea, the iris, the pupil and the lensbut these parts are not in focus and therefore they contribute only tobackground, thus having no considerable interaction to an observer.

In the embodiment of FIG. 1, the aim is to measure a location of a foveaon the detector 106. The fovea sets a one to one correspondence to aneye-gaze on surface, which eye is gazed and a fovea spot is detected andmeasured. A position of the fovea spot on the detector 106 has one toone correspondence to a position on the LCD panel 108. Thus measuring onthe detector 106 a fovea spot, which is eye-gaze, utilizing one to onecorrespondence between the detector 106 and the LCD panel 108, a foveaspot on the LCD panel 108 is formed, which is now an eye-gaze on the LCDpanel 108. The fovea spot appears as dark spot on the picture of thereflected light from the retina 100. Thus an aim is to measure this darkspot. The detector 106 detects and converts detection information todata, which is a numerical or electrical form of detection information.This data is transferred to a controller unit 107. Running software incontrol unit 107 measures and calculates the fovea spot from the dataand transfers this information to the device using the display.

Alternatively, the location of a fovea may be obtained from CCD data,where a CCD detector is used as detector 106. Detection information andthus CCD data is obtained as described above. Also the entire VRD devicecan be corresponding as in the preferred embodiment. The location of afovea from CCD data may be obtained by using a comparison of patterns.From CCD data an initial calibration pattern of blood vessels is formedin control unit 107. After a calibration pattern is set, a newmeasurement and thus a new CCD data of an eye-gaze position is detectedon the CCD detector 106. From new CCD data a pattern of blood vessels isformed in the control unit 107. The control unit performs a series offunctions to implement a pattern recognition calculation of these twopatterns (pattern recognition by comparing a measured pattern with acalibration pattern being known as such). The control unit 107 performsa series of calculations to implement comparison between calibrationpattern and eye-gaze pattern. By these calculations and a comparison (ofthe measured and calibration pattern) the control unit 107 sets thelocation of a fovea spot on the CCD detector 106. Mapping the locationof a fovea, moreover eye-gaze, between the CCD detector 106 and the LCDpanel 108 is implemented correspondingly as was described above, and yetby obtaining the fovea from CCD data using pattern comparison.

FIG. 3 depicts a flow chart of an embodiment of the device and method oftracking eye-gaze according to the invention. A deep blue source 116emits a deep blue or violet light (Step 300). A narrow band bluereflector 118 reflects the emitted light appropriately according tofrequency and direction (Step 302). The emitted and reflected lightreflects back from the eye retina (Step 304). Light reflected from theeye retina reflects from narrow band blue or violet beam splitter 118appropriately according to frequency and direction (Step 306). Adetector 106 detects the reflected light and converts this lightdetection information to electronic or binary data form (Step 308). Thedetector 106 measures a fovea spot from the data (Step 310). The controlunit 107 maps the fovea spot from the detector 106 surface (Step 312).The device implements functions depending on mapped eye-gaze on VRD,e.g. display eye-gaze on LCD panel thus on VRD or set control toeye-gaze location (Step 314).

FIG. 4 depicts a flow chart of an alternative embodiment of the deviceand method of tracking eye-gaze according to the invention. A deep bluesource 116 emits a deep blue or violet light (Step 400). A narrow bandblue reflector 118 reflects the emitted light appropriately according tofrequency and direction (Step 402). The emitted and reflected lightreflects back from the eye retina (Step 404). Light reflected from theeye retina is further reflected from narrow band blue or violet beamsplitter 118 appropriately according to frequency and direction (Step406). A detector 106 detects the light reflected from the narrow bandmirror 118 and converts the light detection information to electronic orbinary data form (Step 408). The control unit 107 calculates a patternrecognition calculation of blood vessels from the data (Step 410). If aninitial calibration pattern does not exists, the calculated pattern isset to form an initial calibration pattern (Steps 411 and 412). Thedevice then sets a new CCD data from streaming eye-gaze information. Ifinitial calibration pattern exists, the control unit 107 compares thecalculation to calibration pattern (Steps 411 and 414). The control unit107 maps the fovea spot from the CCD detector 106 surface to the LCDpanel 108, thus to VRD surface (Step 416). The device implementsfunctions depending on the mapped eye-gaze on VRD, e.g. display eye-gazeon the LCD panel thus on the VRD or sets control to eye-gaze location(Step 418).

Eye-gaze tracking/control should set a data stream delivering a highinformation content to the part of the display that the eye is gazing.Efficient monitoring of eye-gaze is inevitable to eye-gaze controlledcommunication between a human and an electronic device. A control setshould be eye-gaze itself or eye-gaze combined to other existing controlmeans such as a control button. In addition, an eye-gazetracking/control device should provide a response with an imperceptibledelay for a device user for required usage.

FIG. 5 depicts an example of a typical use situation of the eye-gazetracking and controlling device. In FIG. 5, a user or an observer 500utilizes the eye-gaze tracking device integrated into a VRD device 504.A user's eye 502 is used to control VRD device 504.

After eye-gaze is mapped to VRD, VRD device 504 is in standby mode toutilize eye-gaze based controlling to the device used. Eye-gaze controlcan set a data stream delivering a high information content to the partof the display 504 that the eye 502 is gazing. Thus a control set or acontrol master is the eye-gaze itself functioning independently withoutany necessary need for additional control equipment. Also moreover, VRDdisplay is divided into parts, which can provide more information orextra content when eye is gazed to the part of VRD display for apredetermined time period. Time period sets an idle for mapping thecontrol according to part of VRD and prevents false gazes or quickgazes, which can be used to discern a scene of VRD.

Alternatively, a control set or a control master is eye-gaze combined toextra controlling equipment, such as a button, a mouse or a keyboard.After eye-gaze is mapped to VRD, VRD device 504 is in standby mode toutilize eye-gaze based controlling to the device used. Eye-gaze is in apart of VRD display, which user wants to have control, or in a part ofVRD display, which is traditionally used to control the device, userconfirms to utilize eye-gaze control by implementing an action usingextra controlling equipment. Action can be a press of button or akeystroke indicating a mark to device to perform control, which happensin response to the user gazing at a button on the display. An exemplaryangle accuracy is ten minutes of arc.

This paper presents the implementation and embodiments of the inventionwith the help of examples. It is obvious to a person skilled in the art,that the invention is not restricted to details of the embodimentspresented above, and that the invention can be implemented in anotherembodiment without deviating from the characteristics of the invention.Thus, the presented embodiments should be considered illustrative, butnot restricting. Hence, the possibilities of implementing and using theinvention are only restricted by the enclosed patent claims.Consequently, the various options of implementing the invention asdetermined by the claims, including the equivalent implementations, alsobelong to the scope of the present invention.

For example, the mirror and the reflector have been described as havingtransparencies in order to integrate the device. However, it is possibleto design and arrange the device in such a way, that the light fromsources and the picture from the display panel will not meet concretelyand will not disturb one another considerably.

For another example, after eye-gaze has been mapped on a display, thefunction has been for displaying a mark or setting a control. However,various applications can be implemented after mapping eye-gaze. Acontrol can be set to another device, whose part VRD is, e.g.,controlling a mechanical or electronic device utilizing embodiments ofinvention which is a part of the entire device.

What is claimed is:
 1. A method of tracking eye-gaze, the methodcomprising emitting light having a certain wavelength; transferring thelight to the retina of an eye; the certain wavelength of the light beingsuch as to make the fovea of the eye clear, prominent, and resolvable;detecting light that is reflected from said eye to form detectioninformation including the clear, prominent, and resolvable fovea;mapping the detection information to a predetermined surface, saidpredetermined surface being located at a distance from said eye, thelocation of the clear, prominent, and resolvable fovea on saidpredetermined surface forming an eye-gaze point.
 2. The method of claim1, wherein the light has a wavelength in the range of 395 nm to 500 nm.3. The method of claim 1, wherein the light has a wavelength in therange of 395 nm to 430 nm.
 4. The method of claim 1, wherein the lighthas a wavelength of 405 nm.
 5. The method of claim 1, wherein the stepof transferring the light comprises transferring the light withinoptics.
 6. The method of claim 1, wherein the method comprises narrowband beam splitting at least one of the emitted and reflected light. 7.The method of claim 1, wherein the method comprises rotating thepolarization 90 degrees of at least one of the emitted and reflectedlight.
 8. The method of claim 1, wherein said detecting step comprisesconverting said light reflected from said eye retina to electronicinformation on a detector.
 9. The method of claim 1, wherein saidmapping comprises calculating a blood vessels pattern from saiddetection information; comparing said blood vessels pattern to apredetermined pattern; setting the location of the clear, prominent, andresolvable fovea on said predetermined surface based on said comparingstep.
 10. The method of claim 1, further comprising a step of displayinga mark on said predetermined surface at the eye-gaze point.
 11. Themethod of claim 1, further comprising a step of performing a controlfunction relating to the particular point of eye-gaze on thepredetermined surface.
 12. An eye-gaze tracking device, comprising: alight source for emitting light having a certain wavelength; means fortransferring the light to the retina of an eye, the certain wavelengthof the light being such as to make the fovea of the eye clear,prominent, and resolvable; a detector for detecting light that isreflected from said eye to form detection information including theclear, prominent, and resolvable fovea; a surface located at a distancefrom said eye; and means for mapping the detection information to thesurface for locating the clear, prominent, and resolvable fovea on saidsurface for forming an eye-gaze point by the location of the clearprominent, and resolvable fovea.
 13. The device of claim 12, whereinsaid surface includes a display.
 14. The device of claim 12, furthercomprising a virtual reality display comprising a LCD panel.
 15. Thedevice of claim 12, further comprising imaging optics placed betweensaid eye and said surface.
 16. The device of claim 12, furthercomprising a control unit for transferring the detection informationfrom said detector and for converting the detection information to saidsurface.
 17. The device of claim 12, further comprising a narrow bandreflector, a beam splitter and a narrow band mirror for reflecting andfiltering the light at a narrow band in the range of said certainwavelength.
 18. The device of claim 12, wherein said detector is a CCD(Charge Coupled Device) detector.
 19. The device of claim 12, whereinsaid certain wavelength is within the range of 395 nm to 500 nm.
 20. Thedevice of claim 12, wherein said certain wavelength is within the rangeof 395 nm to 430 nm.
 21. The device of claim 12, wherein said certainwavelength is 405 nm.
 22. A mobile phone having an eye-gaze trackingdevice, comprising: a light source for emitting light falling within acertain wavelength range; means for transferring the light to the retinaof an eye, the certain wavelength of the light being such as to make thefovea of the eye clear, prominent, and resolvable; a detector fordetecting light that is reflected from said eye to form detectioninformation including the clear, prominent, and resolvable fovea; adisplay located at a distance from said eye; and means for mapping thedetection information to the display for locating the clear, prominent,and resolvable fovea on said display for forming an eye-gaze point bythe location of the clear, prominent, and resolvable fovea.
 23. Themobile phone having an eye-gaze device of claim 22, wherein the certainwavelength range is a narrowband blue wavelength range.